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Abstract Body 


Background / Context: 

Academic achievement in the nation’s lowest performing schools is troubling. Despite 
improvements in students’ overall academic achievement over the past few decades, proficiency 
gaps in reading and mathematics remain across income, racial, and ethnic groups’. One 
promising strategy for reducing achievement gaps is to expand the school day, often called 
Expanded Learning Time (ELT); evidence suggests that students’ increased access—through 
ELT programming—to enrichment activities can improve their academic motivation and social- 
emotional skill development. 


Since 1995, Citizen Schools (CS) has developed and implemented its own ELT model. CS 
partners with middle schools serving predominantly low income, racial or ethnic minority, and 
academically struggling students. The CS model relies upon an additional shift of educators and 
community volunteers to engage middle school students in hands-on apprenticeships, while 
simultaneously providing individualized supports to ensure academic and future success. 


Several recent meta-analyses have examined studies of various ELT models, prioritizing studies 
based on stronger research designs, and findings are mixed. Some research suggests that 
expanded learning time improves non-academic student outcomes such as students’ academic 
motivation, socio-emotional skill development, behavior management, school bonding, and 
positive self-perceptions, while others found null effects”. 


Purpose: 

The study was designed to address questions about implementation to inform program 
improvement as well as questions about the impact on student engagement, aspirations, and 
achievement. 


Setting and Sample: 

Middle schools in MA, NY, NJ, TX, IL, NM, CA serving predominantly low-income, racial or 
ethnic minority, and academically struggling students. All students in grades designated for the 
ELT program are required to participate. 


* National Center for Education Statistics. (2013). The nation's report card: Trends in academic progress 2012 
(NCES 2013-456). Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. 
Department of Education. 

Zief, S. G., Lauver, S., & Maynard, R. (2006). The impacts of afterschool programs on student outcomes. Campbell 
Systematic Reviews, 3. 

5s Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of afterschool programs that seek to promote 
personal and social skills in children and adolescents. American Journal of Community Psychology, 45(3/4), 294— 
309. 

5 Kidron, Y, & Lindsay, J. (2014). The effects of increased learning time on student academic and nonacademic 
outcomes: Findings from a meta analytic review. Prepared by American Institutes for Research for the Institute for 
Education Sciences, US Department of Education. 

Redd, A., Boccanfuso, C., Walker, K., Princiotta, D., Knewstub, D., & Moore, K. (2012). Expanding time for learning 
both inside and outside the classroom: A Review of the evidence bases. Prepared by Child Trends for the Wallace 
Foundation. 
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Intervention: 

CS ELT mobilizes a second shift of educators for two to three hours per day (four days/week), 
who provide academic support, leadership development, and “apprenticeships” at the end of the 
school day. 


Research Design: 

The five-year evaluation included implementation and impact components, and incorporated 
surveys, interviews, and site visits to assess how schools integrate ELT into their school days, as 
well as relying upon comparison schools—matched on key pre-intervention observable 
characteristics—to estimate impacts. Analyses examining student achievement, where baseline 
data are available, used a comparative interrupted time series (C-ITS) design that combined pre- 
program data and data from matched comparison (MC) schools to produce estimated effects. 


Two unique elements of the research design included 1) the creation of an implementation index 
to systematically measure fidelity to the CS model at each school, and 2) the ability to link the 
implementation index measure to the achievement data to explain variation in student 
achievement. 


Data Collection and Analysis: 


Study Year 
School Year 
1 2 3 4 5 
Data Collection Activity 2010-11 2011-12 2012-13 2013-14 2014-15 
Site visit (CS ELT only) v v v 
Principal Telephone Interview (CS ELT) v v v v 
Principal Telephone Interview (MC) v v v 
Campus Director Telephone Interview v v v x 
Survey 
Student (CS ELT & MC) v v 
Teacher (CS ELT & MC) v v v 
CS staff (CS ELT only) v v v 
Extant (test score) data v v v v 


Interview data were synthesized, cleaned and standardized before being uploaded into a 
qualitative data analysis software package. Further, the study team developed a numeric 
implementation index keyed to core principles of effective ELT implementation. 


CS ELT participants’ survey responses were compared to responses from participants in matched 
comparison schools using an HLM model. Achievement data were assessed using a C-SITS 
design which modeled CS ELT as an “interruption” in what would otherwise be assumed to be 
somewhat stable levels of a particular outcome. Models included up to five years of baseline 
data, along with school and year fixed effects. 
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Findings: 
e How CS selected appropriate partner schools, planned and communicated about ELT 
with stakeholders, and anticipated logistical and programmatic challenges to the model 
demonstrably influenced the ELT implementation. 


e CS programming occurs primarily through its staff; consequently, recruitment, hiring, 
preparation and support of staff have clear ramifications for how the program operates 
and is perceived. 


e The two most prevalent concerns about program sustainability include financial 
sustainability and campus stability. 


e The majority of schools implemented CS ELT with moderate levels of fidelity. 


e CS students were significantly more likely to report that their peers were positively 
engaged, that the ELT program helped their self-esteem and pro-social behaviors, and 
that they participated in activities to help them learn about college and careers than their 
counterparts. However, there were also some ongoing challenges (perceptions of CS 
staff). 


e Data on student achievement in English/Language Arts (ELA) and math indicated no 
overall statistically significant impacts of CS ELT and no difference between higher and 
lower implementing schools; exploratory analyses suggest a significant positive impact 
on math achievement in the first year of implementation and a marginally significant 
positive effect of CS ELT on ha grade math achievement. 


Conclusions: 


Schools varied considerably in how the model is incorporated into school contexts, yet 
variability seems to be essential for the model to be adaptable across such diverse settings. The 
implementation findings also highlight some of the challenges associated with launching a multi- 
faceted model in dynamic settings, coupled with built-in staffing changes. 


The outcome-focused findings are mixed. The study finds positive effects on student engagement 
and aspirations and negative perceptions about students’ CS ELT experiences. The confirmatory 
findings indicate no overall significant impact on student performance, as measured by 
standardized achievement test scores in ELA and math, and no significant differences in 
achievement between higher and lower implementing schools. Exploratory findings indicate a 
significant positive impact of CS ELT on math achievement in the first year of implementation 
and a marginally significant positive effect of CSELT on 7th grade math achievement. 


Prior research on ELT interventions is mixed. Interventions designed to improve academic 
achievement, such as KIPP, or Higher Achievement, are more likely to affect academic 
outcomes than interventions designed to broaden students’ enrichment or social-emotional 
experiences. The CS ELT model emphasizes non-academic learning opportunities hypothesized 
to be necessary precursors to improved achievement. Perhaps student engagement and 
aspirational short-term outcomes are more appropriate outcomes on which to focus, given CS’ 
emphasis on team-building and exposing students to novel, hands-on, real-world experiences 
through apprenticeships. 
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